
 <!DOCTYPE HTML>
<html lang="zh-CN">
<head>
  <meta charset="UTF-8">
  
    <title>使用 Python 轻松抓取网页 | Jark&#39;s Blog</title>
    <meta name="viewport" content="width=device-width, initial-scale=1,user-scalable=no">
    
    <meta name="author" content="WuChong">
    
    <meta name="description" itemprop="description" content="翻译自 Miguel 写的一篇 Python 爬虫入门教学。以一个很有趣的目标为驱动，一步一步教你如何抓取网页，浅显易懂，非常适合初学者。知识点涉及网页下载、信息抽取、多进程等。由于下载的页面被墙，加入通过设置代理让爬虫翻墙的章节。">
    
    
    <meta name="description" content="翻译自 Miguel 写的一篇 Python 爬虫入门教学。以一个很有趣的目标为驱动，一步一步教你如何抓取网页，浅显易懂，非常适合初学者。知识点涉及网页下载、信息抽取、多进程等。由于下载的页面被墙，加入通过设置代理让爬虫翻墙的章节。">
<meta property="og:type" content="article">
<meta property="og:title" content="使用 Python 轻松抓取网页">
<meta property="og:url" content="http://wuchong.me/blog/2014/04/24/easy-web-scraping-with-python/">
<meta property="og:site_name" content="Jark's Blog">
<meta property="og:description" content="翻译自 Miguel 写的一篇 Python 爬虫入门教学。以一个很有趣的目标为驱动，一步一步教你如何抓取网页，浅显易懂，非常适合初学者。知识点涉及网页下载、信息抽取、多进程等。由于下载的页面被墙，加入通过设置代理让爬虫翻墙的章节。">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="使用 Python 轻松抓取网页">
<meta name="twitter:description" content="翻译自 Miguel 写的一篇 Python 爬虫入门教学。以一个很有趣的目标为驱动，一步一步教你如何抓取网页，浅显易懂，非常适合初学者。知识点涉及网页下载、信息抽取、多进程等。由于下载的页面被墙，加入通过设置代理让爬虫翻墙的章节。">
<meta name="twitter:creator" content="@jarkwu">
<link rel="publisher" href="111190881341800841449">

    
    <link rel="alternative" href="/atom.xml" title="Jark&#39;s Blog" type="application/atom+xml">
    
    
    <link rel="icon" href="/img/favicon.ico">
    
    
    <link rel="apple-touch-icon" href="/img/jacman.jpg">
    <link rel="apple-touch-icon-precomposed" href="/img/jacman.jpg">
    
    <link rel="stylesheet" href="/css/style.css" type="text/css">
</head>

  <body>
    <header>
      <div>
		
			<div id="imglogo">
				<a href="/"><img src="/img/logo.png" alt="Jark&#39;s Blog" title="Jark&#39;s Blog"/></a>
			</div>
			
			<div id="textlogo">
				<h1 class="site-name"><a href="/" title="Jark&#39;s Blog">Jark&#39;s Blog</a></h1>
				<h2 class="blog-motto">当你的才华还撑不起你的野心时，你就应该静下心来学习。</h2>
			</div>
			<div class="navbar"><a class="navbutton navmobile" href="#" title="菜单">
			</a></div>
			<nav class="animated">
				<ul>
					<ul>
					 
						<li><a href="/">首页</a></li>
					
						<li><a href="/archives">归档</a></li>
					
						<li><a href="/about">关于</a></li>
					
					<li>
 					
						<form class="search" action="http://zhannei.baidu.com/cse/search" target="_blank">
							<label>Search</label>
						<input name="s" type="hidden" value= 783281470518440642 ><input type="text" name="q" size="30" placeholder="搜索"><br>
						</form>
					
					</li>
				</ul>
			</nav>			
</div>
    </header>
    <div id="container">
      <div id="main" class="post" itemscope itemprop="blogPost">
  
	<article itemprop="articleBody"> 
		<header class="article-info clearfix">
  <h1 itemprop="name">
    
      <a href="/blog/2014/04/24/easy-web-scraping-with-python/" title="使用 Python 轻松抓取网页" itemprop="url">使用 Python 轻松抓取网页</a>
  </h1>
  <p class="article-author">By
       
		<a href="https://plus.google.com/111190881341800841449?rel=author" title="WuChong" target="_blank" itemprop="author">WuChong</a>
		
  <p class="article-time">
    <time datetime="2014-04-24T15:04:00.000Z" itemprop="datePublished"> 发表于 2014-04-24</time>
    
  </p>
</header>
	<div class="article-content">
		
		<div id="toc" class="toc-article">
			<strong class="toc-title">文章目录</strong>
		
			<ol class="toc"><li class="toc-item toc-level-2"><a class="toc-link" href="#问题"><span class="toc-number">1.</span> <span class="toc-text">问题</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#工具"><span class="toc-number">2.</span> <span class="toc-text">工具</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#抓取相连页面"><span class="toc-number">3.</span> <span class="toc-text">抓取相连页面</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#并行处理"><span class="toc-number">4.</span> <span class="toc-text">并行处理</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#完成爬虫脚本"><span class="toc-number">5.</span> <span class="toc-text">完成爬虫脚本</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#结论"><span class="toc-number">6.</span> <span class="toc-text">结论</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#爬虫翻墙"><span class="toc-number">7.</span> <span class="toc-text">爬虫翻墙</span></a></li></ol>
		
		</div>
		
		<p>[ 翻译自英文原文：<a href="http://blog.miguelgrinberg.com/post/easy-web-scraping-with-python" target="_blank" rel="external">Easy Web Scraping with Python</a> ]</p>
<p>一年多以前我写了一篇文章<a href="http://blog.miguelgrinberg.com/post/easy-web-scraping-with-nodejs" target="_blank" rel="external">「web scraping using Node.js」</a>。今天我重新回顾了这个话题，但是这一次我将使用 Python，这样这两种语言所提供的技术就能进行对比和比较。</p>
<h2 id="问题">问题</h2>
<p>我敢肯定你知道，我在本月初参加了在蒙特利尔举办的 PyCon 大会。所有的演讲和教程的视频都已经发布到 YouTube 上了，目录在 <a href="http://pyvideo.org/category/50/pycon-us-2014" target="_blank" rel="external">pyvideo.org</a>。</p>
<p>我认为知道这个大会上的哪些视频最受欢迎将会是非常有用的，所以我们将要写一个爬虫脚本。这个脚本将会从 pyvideo.org 上获取有效的视频列表，然后从每个 YouTube 页面获取观看的统计数据。听起来很有趣？让我们开始吧！</p>
<h2 id="工具">工具</h2>
<p>在抓取网站中有两个基本的任务：</p>
<ol>
<li>加载网页到一个 string 里。</li>
<li>从网页中解析 HTML 来定位感兴趣的位置。</li>
</ol>
<p>Python 为上面两个任务提供了两个超棒的工具。我将使用 <a href="http://docs.python-requests.org/en/latest/" target="_blank" rel="external">requests</a> 去加载网页，用 <a href="http://www.crummy.com/software/BeautifulSoup/bs4/doc/" target="_blank" rel="external">BeautifulSoup</a> 去做解析。</p>
<p>我们可以把上面两个包放到一个虚拟环境：</p>
<figure class="highlight"><table><tr><td class="gutter"><pre>1
2
3
4
</pre></td><td class="code"><pre><span class="variable">$ </span>mkdir pycon-scraper
<span class="variable">$ </span>virtualenv venv
<span class="variable">$ </span>source venv/bin/activate
(venv) <span class="variable">$ </span>pip install requests beautifulsoup4
</pre></td></tr></table></figure><br>如果使用的是 Windows 操作系统，注意上面虚拟环境的激活命令是不同的，你应该使用<code>venv\Scripts\activate</code>。<br><a id="more"></a><br>##基本的抓取技术<br>在写一个爬虫脚本时，第一件事情就是手动观察要抓取的页面来确定数据如何定位。<br><br>首先，我们要看一看在 <a href="http://pyvideo.org/category/50/pycon-us-2014" target="_blank" rel="external">http://pyvideo.org/category/50/pycon-us-2014</a> 上的 PyCon 大会视频列表。检查这个页面的 HTML 源代码我们发现视频列表的结果差不多是长这样的：<br><br><figure class="highlight HTML"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="code"><pre><span class="tag">&lt;<span class="title">div</span> <span class="attribute">id</span>=<span class="value">"video-summary-content"</span>&gt;</span>
    <span class="tag">&lt;<span class="title">div</span> <span class="attribute">class</span>=<span class="value">"video-summary"</span>&gt;</span>    <span class="comment">&lt;!-- first video --&gt;</span>
        <span class="tag">&lt;<span class="title">div</span> <span class="attribute">class</span>=<span class="value">"thumbnail-data"</span>&gt;</span>...<span class="tag">&lt;/<span class="title">div</span>&gt;</span>
        <span class="tag">&lt;<span class="title">div</span> <span class="attribute">class</span>=<span class="value">"video-summary-data"</span>&gt;</span>
            <span class="tag">&lt;<span class="title">div</span>&gt;</span>
                <span class="tag">&lt;<span class="title">strong</span>&gt;</span><span class="tag">&lt;<span class="title">a</span> <span class="attribute">href</span>=<span class="value">"#link to video page#"</span>&gt;</span>#title#<span class="tag">&lt;/<span class="title">a</span>&gt;</span><span class="tag">&lt;/<span class="title">strong</span>&gt;</span>
            <span class="tag">&lt;/<span class="title">div</span>&gt;</span>
        <span class="tag">&lt;/<span class="title">div</span>&gt;</span>
    <span class="tag">&lt;/<span class="title">div</span>&gt;</span>
    <span class="tag">&lt;<span class="title">div</span> <span class="attribute">class</span>=<span class="value">"video-summary"</span>&gt;</span>    <span class="comment">&lt;!-- second video --&gt;</span>
        ...
    <span class="tag">&lt;/<span class="title">div</span>&gt;</span>
    ...
<span class="tag">&lt;/<span class="title">div</span>&gt;</span>
</pre></td></tr></table></figure>

<p>那么第一个任务就是加载这个页面，然后抽取每个单独页面的链接，因为到 YouTube 视频的链接都在这些单独页面上。</p>
<p>使用<code>requests</code>来加载一个 web 页面是非常简单的：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
</pre></td><td class="code"><pre><span class="keyword">import</span> requests
response = requests.get(<span class="string">'http://pyvideo.org/category/50/pycon-us-2014'</span>)
</pre></td></tr></table></figure>

<p>就是它！在这个函数返回后就能从<code>response.text</code>中获得这个页面的 HTML 。</p>
<p>下一个任务是抽取每一个单独视频页面的链接。通过 BeautifulSoup 使用 CSS 选择器语法就能完成它，如果你是客户端开发者的话你可能对这会很熟悉。</p>
<p>为了获得这些链接，我们要使用一个选择器，它能抓取在每一个 id 为<code>video-summary-data</code>的<code>&lt;div&gt;</code>中所有的<code>&lt;a&gt;</code>元素。由于每个视频都有几个<code>&lt;a&gt;</code>元素，我们将只保留那些 URL 以<code>/video</code>开头的<code>&lt;a&gt;</code>元素，这些就是唯一的单独视频页面。实现上述标准的 CSS 选择器是<code>div.video-summary-data a[href^=/video]</code>。下面的代码片段通过 BeautifulSoup 使用这个选择器来获得指向视频页面的<code>&lt;a&gt;</code>元素：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
</pre></td><td class="code"><pre><span class="keyword">import</span> bs4
soup = bs4.BeautifulSoup(response.text)
links = soup.select(<span class="string">'div.video-summary-data a[href^=/video]'</span>)
</pre></td></tr></table></figure>

<p>因为我们真正关心的是这个链接本身而不是包含它的<code>&lt;a&gt;</code>元素，我们可以使用列表解析来改善上述代码。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
</pre></td><td class="code"><pre>links = [a.attrs.get(<span class="string">'href'</span>) <span class="keyword">for</span> a <span class="keyword">in</span> soup.select(<span class="string">'div.video-summary-data a[href^=/video]'</span>)]
</pre></td></tr></table></figure>

<p>现在，我们已经有了一个包含所有链接的数组，这些链接指向了每个单独页面。</p>
<p>下面这段脚本整理了目前我们提到的所有技术：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="code"><pre><span class="keyword">import</span> requests
<span class="keyword">import</span> bs4

root_url = <span class="string">'http://pyvideo.org'</span>
index_url = root_url + <span class="string">'/category/50/pycon-us-2014'</span>

<span class="function"><span class="keyword">def</span> <span class="title">get_video_page_urls</span><span class="params">()</span>:</span>
    response = requests.get(index_url)
    soup = bs4.BeautifulSoup(response.text)
    <span class="keyword">return</span> [a.attrs.get(<span class="string">'href'</span>) <span class="keyword">for</span> a <span class="keyword">in</span> soup.select(<span class="string">'div.video-summary-data a[href^=/video]'</span>)]

print(get_video_page_urls())
</pre></td></tr></table></figure>

<p>如果你运行上面这段脚本你将会获得一个满是 URL 的数组。现在我们需要去解析每个 URL 以获得更多关于每场 PyCon 会议的信息。</p>
<h2 id="抓取相连页面">抓取相连页面</h2>
<p>下一步是加载我们的 URL 数组中每一个页面。如果你想要看看这些页面长什么样的话，这儿是个样例：<a href="http://pyvideo.org/video/2668/writing-restful-web-services-with-flask" target="_blank" rel="external">http://pyvideo.org/video/2668/writing-restful-web-services-with-flask</a>。没错，那就是我，那是我会议中的一个！</p>
<p>从这些页面我们可以抓取到会议的标题，在页面的顶部能看到它。我们也可以从侧边栏获得演讲者的姓名和 YouTube 的链接，侧边栏在嵌入视频的右下方。获取这些元素的代码展示在下方：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
7
</pre></td><td class="code"><pre><span class="function"><span class="keyword">def</span> <span class="title">get_video_data</span><span class="params">(video_page_url)</span>:</span>
    video_data = {}
    response = requests.get(root_url + video_page_url)
    soup = bs4.BeautifulSoup(response.text)
    video_data[<span class="string">'title'</span>] = soup.select(<span class="string">'div#videobox h3'</span>)[<span class="number">0</span>].get_text()
    video_data[<span class="string">'speakers'</span>] = [a.get_text() <span class="keyword">for</span> a <span class="keyword">in</span> soup.select(<span class="string">'div#sidebar a[href^=/speaker]'</span>)]
    video_data[<span class="string">'youtube_url'</span>] = soup.select(<span class="string">'div#sidebar a[href^=http://www.youtube.com]'</span>)[<span class="number">0</span>].get_text()
</pre></td></tr></table></figure>

<p>关于这个函数需要注意的一些事情：</p>
<ul>
<li>从首页抓取的 URL 是相对路径，所以<code>root_url</code>需要加到前面。</li>
<li>大会标题是从 id 为<code>videobox</code>的<code>&lt;div&gt;</code>里的<code>&lt;h3&gt;</code>元素中获得的。注意<code>[0]</code>是必须的，因为调用<code>select()</code>返回的是一个数组，即使只有一个匹配。</li>
<li>演讲者的姓名和 YouTube 链接的获取方式与首页上的链接获取方式类似。</li>
</ul>
<p>现在就剩下从每个视频的 YouTube 页面抓取观看数了。接着上面的函数写下去其实是非常简单的。同样，我们也可以抓取 like 数和 dislike 数。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="code"><pre><span class="function"><span class="keyword">def</span> <span class="title">get_video_data</span><span class="params">(video_page_url)</span>:</span>
    <span class="comment"># ...</span>
    response = requests.get(video_data[<span class="string">'youtube_url'</span>])
    soup = bs4.BeautifulSoup(response.text)
    video_data[<span class="string">'views'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>,
                                     soup.select(<span class="string">'.watch-view-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    video_data[<span class="string">'likes'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>,
                                     soup.select(<span class="string">'.likes-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    video_data[<span class="string">'dislikes'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>, 
                                        soup.select(<span class="string">'.dislikes-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    <span class="keyword">return</span> video_data
</pre></td></tr></table></figure>

<p>上述调用<code>soup.select()</code>函数，使用指定了 id 名字的选择器，采集到了视频的统计数据。但是元素的文本需要被处理一下才能变成数字。考虑观看数的例子，在 YouTube 上显示的是<code>&quot;1,344 views&quot;</code>。用一个空格分开（split）数字和文本后，只有第一部分是有用的。由于数字里有逗号，可以用正则表达式过滤掉任何不是数字的字符。</p>
<p>为了完成爬虫，下面的函数调用了之前提到的所有代码：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
</pre></td><td class="code"><pre><span class="function"><span class="keyword">def</span> <span class="title">show_video_stats</span><span class="params">()</span>:</span>
    video_page_urls = get_video_page_urls()
    <span class="keyword">for</span> video_page_url <span class="keyword">in</span> video_page_urls:
        <span class="keyword">print</span> get_video_data(video_page_url)
</pre></td></tr></table></figure>


<h2 id="并行处理">并行处理</h2>
<p>上面到目前为止的脚本工作地很好，但是有一百多个视频它就要跑个一会儿了。事实上我们没做什么工作，大部分时间都浪费在了下载页面上，在这段时间脚本时被阻塞的。如果脚本能同时跑多个下载任务，可能就会更高效了，是吗？</p>
<p>回顾当时写一篇使用 Node.js 的爬虫文章的时候，并发性是伴随 JavaScript 的异步特性自带来的。使用 Python 也能做到，不过需要显示地指定一下。像这个例子，我将开启一个拥有8个可并行化进程的进程池。代码出人意料的简洁：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
</pre></td><td class="code"><pre><span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Pool

<span class="function"><span class="keyword">def</span> <span class="title">show_video_stats</span><span class="params">(options)</span>:</span>
    pool = Pool(<span class="number">8</span>)
    video_page_urls = get_video_page_urls()
    results = pool.map(get_video_data, video_page_urls)
</pre></td></tr></table></figure>

<p><code>multiprocessing.Pool</code> 类开启了8个工作进程等待分配任务运行。为什么是8个？这是我电脑上核数的两倍。当时实验不同大小的进程池时，我发现这是最佳的大小。小于8个使脚本跑的太慢，多于8个也不会让它更快。</p>
<p>调用<code>pool.map()</code>类似于调用常规的<code>map()</code>，它将会对第二个参数指定的迭代变量中的每个元素调用一次第一个参数指定的函数。最大的不同是，它将发送这些给进程池所拥有的进程运行，所以在这个例子中八个任务将会并行运行。</p>
<p>节省下来的时间是相当大的。在我的电脑上，第一个版本的脚本用了75秒完成，然而进程池的版本做了同样的工作只用了16秒！</p>
<h2 id="完成爬虫脚本">完成爬虫脚本</h2>
<p>我最终版本的爬虫脚本在获得数据后还做了更多的事情。</p>
<p>我添加了一个<code>--sort</code>命令行参数去指定一个排序标准，可以指定views，likes或者dislikes。脚本将会根据指定属性对结果数组进行递减排序。另一个参数，<code>--max</code>代表了要显示的结果数的个数，万一你只想看排名靠前的几条而已。最后，我还添加了一个<code>--csv</code>选项，为了可以轻松地将数据导到电子制表软件中，可以指定数据以 CSV 格式打印出来，而不是表对齐格式。</p>
<p>完整脚本显示在下方：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
</pre></td><td class="code"><pre><span class="keyword">import</span> argparse
<span class="keyword">import</span> re
<span class="keyword">from</span> multiprocessing <span class="keyword">import</span> Pool
<span class="keyword">import</span> requests
<span class="keyword">import</span> bs4

root_url = <span class="string">'http://pyvideo.org'</span>
index_url = root_url + <span class="string">'/category/50/pycon-us-2014'</span>

<span class="function"><span class="keyword">def</span> <span class="title">get_video_page_urls</span><span class="params">()</span>:</span>
    response = requests.get(index_url)
    soup = bs4.BeautifulSoup(response.text)
    <span class="keyword">return</span> [a.attrs.get(<span class="string">'href'</span>) <span class="keyword">for</span> a <span class="keyword">in</span> soup.select(<span class="string">'div.video-summary-data a[href^=/video]'</span>)]

<span class="function"><span class="keyword">def</span> <span class="title">get_video_data</span><span class="params">(video_page_url)</span>:</span>
    video_data = {}
    response = requests.get(root_url + video_page_url)
    soup = bs4.BeautifulSoup(response.text)
    video_data[<span class="string">'title'</span>] = soup.select(<span class="string">'div#videobox h3'</span>)[<span class="number">0</span>].get_text()
    video_data[<span class="string">'speakers'</span>] = [a.get_text() <span class="keyword">for</span> a <span class="keyword">in</span> soup.select(<span class="string">'div#sidebar a[href^=/speaker]'</span>)]
    video_data[<span class="string">'youtube_url'</span>] = soup.select(<span class="string">'div#sidebar a[href^=http://www.youtube.com]'</span>)[<span class="number">0</span>].get_text()
    response = requests.get(video_data[<span class="string">'youtube_url'</span>])
    soup = bs4.BeautifulSoup(response.text)
    video_data[<span class="string">'views'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>,
                                     soup.select(<span class="string">'.watch-view-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    video_data[<span class="string">'likes'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>,
                                     soup.select(<span class="string">'.likes-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    video_data[<span class="string">'dislikes'</span>] = int(re.sub(<span class="string">'[^0-9]'</span>, <span class="string">''</span>,
                                        soup.select(<span class="string">'.dislikes-count'</span>)[<span class="number">0</span>].get_text().split()[<span class="number">0</span>]))
    <span class="keyword">return</span> video_data

<span class="function"><span class="keyword">def</span> <span class="title">parse_args</span><span class="params">()</span>:</span>
    parser = argparse.ArgumentParser(description=<span class="string">'Show PyCon 2014 video statistics.'</span>)
    parser.add_argument(<span class="string">'--sort'</span>, metavar=<span class="string">'FIELD'</span>, choices=[<span class="string">'views'</span>, <span class="string">'likes'</span>, <span class="string">'dislikes'</span>],
                        default=<span class="string">'views'</span>,
                        help=<span class="string">'sort by the specified field. Options are views, likes and dislikes.'</span>)
    parser.add_argument(<span class="string">'--max'</span>, metavar=<span class="string">'MAX'</span>, type=int, help=<span class="string">'show the top MAX entries only.'</span>)
    parser.add_argument(<span class="string">'--csv'</span>, action=<span class="string">'store_true'</span>, default=<span class="keyword">False</span>,
                        help=<span class="string">'output the data in CSV format.'</span>)
    parser.add_argument(<span class="string">'--workers'</span>, type=int, default=<span class="number">8</span>,
                        help=<span class="string">'number of workers to use, 8 by default.'</span>)
    <span class="keyword">return</span> parser.parse_args()

<span class="function"><span class="keyword">def</span> <span class="title">show_video_stats</span><span class="params">(options)</span>:</span>
    pool = Pool(options.workers)
    video_page_urls = get_video_page_urls()
    results = sorted(pool.map(get_video_data, video_page_urls), key=<span class="keyword">lambda</span> video: video[options.sort],
                     reverse=<span class="keyword">True</span>)
    max = options.max
    <span class="keyword">if</span> max <span class="keyword">is</span> <span class="keyword">None</span> <span class="keyword">or</span> max &gt; len(results):
        max = len(results)
    <span class="keyword">if</span> options.csv:
        print(<span class="string">u'"title","speakers", "views","likes","dislikes"'</span>)
    <span class="keyword">else</span>:
        print(<span class="string">u'Views  +1  -1 Title (Speakers)'</span>)
    <span class="keyword">for</span> i <span class="keyword">in</span> range(max):
        <span class="keyword">if</span> options.csv:
            print(<span class="string">u'"{0}","{1}",{2},{3},{4}'</span>.format(
                results[i][<span class="string">'title'</span>], <span class="string">', '</span>.join(results[i][<span class="string">'speakers'</span>]), results[i][<span class="string">'views'</span>],
                results[i][<span class="string">'likes'</span>], results[i][<span class="string">'dislikes'</span>]))
        <span class="keyword">else</span>:
            print(<span class="string">u'{0:5d} {1:3d} {2:3d} {3} ({4})'</span>.format(
                results[i][<span class="string">'views'</span>], results[i][<span class="string">'likes'</span>], results[i][<span class="string">'dislikes'</span>], results[i][<span class="string">'title'</span>],
                <span class="string">', '</span>.join(results[i][<span class="string">'speakers'</span>])))

<span class="keyword">if</span> __name__ == <span class="string">'__main__'</span>:
    show_video_stats(parse_args())
</pre></td></tr></table></figure>

<p>下方输出的是在我写完代码时前25个观看数最多的会议：</p>
<pre><code>(venv) $ python pycon-scraper.py <span class="comment">--sort views --max 25 --workers 8</span>
Views  +<span class="number">1</span>  -<span class="number">1</span> Title (Speakers)
 <span class="number">3002</span>  <span class="number">27</span>   <span class="number">0</span> Keynote - Guido Van Rossum (Guido Van Rossum)
 <span class="number">2564</span>  <span class="number">21</span>   <span class="number">0</span> Computer science fundamentals <span class="keyword">for</span> self-taught programmers (Justin Abrahms)
 <span class="number">2369</span>  <span class="number">17</span>   <span class="number">0</span> Ansible - Python-Powered Radically Simple IT Automation (Michael Dehaan)
 <span class="number">2165</span>  <span class="number">27</span>   <span class="number">6</span> Analyzing Rap Lyrics <span class="keyword">with</span> Python (Julie Lavoie)
 <span class="number">2158</span>  <span class="number">24</span>   <span class="number">3</span> Exploring Machine Learning <span class="keyword">with</span> Scikit-learn (Jake Vanderplas, Olivier Grisel)
 <span class="number">2065</span>  <span class="number">13</span>   <span class="number">0</span> Fast Python, Slow Python (Alex Gaynor)
 <span class="number">2024</span>  <span class="number">24</span>   <span class="number">0</span> Getting Started <span class="keyword">with</span> Django, a crash course (Kenneth Love)
 <span class="number">1986</span>  <span class="number">47</span>   <span class="number">0</span> It<span class="attribute">'s</span> Dangerous <span class="keyword">to</span> Go Alone: Battling the Invisible Monsters <span class="keyword">in</span> Tech (Julie Pagano)
 <span class="number">1843</span>  <span class="number">24</span>   <span class="number">0</span> Discovering Python (David Beazley)
 <span class="number">1672</span>  <span class="number">22</span>   <span class="number">0</span> <span class="keyword">All</span> Your Ducks <span class="keyword">In</span> A Row: Data Structures <span class="keyword">in</span> the Standard <span class="keyword">Library</span> <span class="keyword">and</span> Beyond (Brandon Rhodes)
 <span class="number">1558</span>  <span class="number">17</span>   <span class="number">1</span> Keynote - Fernando Pérez (Fernando Pérez)
 <span class="number">1449</span>   <span class="number">6</span>   <span class="number">0</span> Descriptors <span class="keyword">and</span> Metaclasses - Understanding <span class="keyword">and</span> Using Python<span class="attribute">'s</span> More Advanced Features (Mike Müller)
 <span class="number">1402</span>  <span class="number">12</span>   <span class="number">0</span> Flask by Example (Miguel Grinberg)
 <span class="number">1342</span>   <span class="number">6</span>   <span class="number">0</span> Python Epiphanies (Stuart Williams)
 <span class="number">1219</span>   <span class="number">5</span>   <span class="number">0</span> <span class="number">0</span> <span class="keyword">to</span> <span class="number">00111100</span> <span class="keyword">with</span> web2py (G. Clifford Williams)
 <span class="number">1169</span>  <span class="number">18</span>   <span class="number">0</span> Cheap Helicopters <span class="keyword">In</span> My Living Room (Ned Jackson Lovely)
 <span class="number">1146</span>  <span class="number">11</span>   <span class="number">0</span> IPython <span class="keyword">in</span> depth: high productivity interactive <span class="keyword">and</span> parallel python (Fernando Perez)
 <span class="number">1127</span>   <span class="number">5</span>   <span class="number">0</span> <span class="number">2</span>D/<span class="number">3</span>D graphics <span class="keyword">with</span> Python <span class="keyword">on</span> mobile platforms (Niko Skrypnik)
 <span class="number">1081</span>   <span class="number">8</span>   <span class="number">0</span> Generators: The Final Frontier (David Beazley)
 <span class="number">1067</span>  <span class="number">12</span>   <span class="number">0</span> Designing Poetic APIs (Erik Rose)
 <span class="number">1064</span>   <span class="number">6</span>   <span class="number">0</span> Keynote - John Perry Barlow (John Perry Barlow)
 <span class="number">1029</span>  <span class="number">10</span>   <span class="number">0</span> What <span class="keyword">Is</span> Async, How Does It Work, <span class="keyword">And</span> <span class="keyword">When</span> Should I <span class="keyword">Use</span> It? (A. Jesse Jiryu Davis)
  <span class="number">981</span>  <span class="number">11</span>   <span class="number">0</span> The Sorry State <span class="keyword">of</span> SSL (Hynek Schlawack)
  <span class="number">961</span>  <span class="number">12</span>   <span class="number">2</span> Farewell <span class="keyword">and</span> Welcome Home: Python <span class="keyword">in</span> Two Genders (Naomi Ceder)
  <span class="number">958</span>   <span class="number">6</span>   <span class="number">0</span> Getting Started Testing (Ned Batchelder)
</code></pre><h2 id="结论">结论</h2>
<p>我希望这篇文章作为使用 Python 抓取网页的入门介绍对你是有帮助的。我在使用 Python 的过程中一直很惊喜，这些工具健壮又强大，而且相比于 JavaScript 的一个事实是异步调优可以放在最后，而对于 JavaScript 你不可能避免从一开始就工作在异步模式下</p>
<p>————————————————————————- 原 文 完 ——————————————————————————</p>
<h2 id="爬虫翻墙">爬虫翻墙</h2>
<p>但是如果我们直接运行上面那段最终代码的话是妥妥的会超时报错的。原因你肯定知道，就是不可逾越的长城。一般 Pyvideo.org 是能上去，但是 YouTube 被墙了。所以这里就需要代理出马了。最简单的可以安装 <a href="https://code.google.com/p/goagent/wiki/InstallGuide" target="_blank" rel="external">GoAgent</a> 。在本地运行 GoAgent 后，使用 <a href="http://docs.python-requests.org/en/latest/user/advanced/#proxies" target="_blank" rel="external">requests.get()</a> 的 proxies 参数，将代理设置成本地的 127.0.0.1 端口为 8087 ，爬虫就能够通过代理访问网页了。使用的代码如下：</p>
<p><figure class="highlight python"><table><tr><td class="gutter"><pre>1
2
</pre></td><td class="code"><pre>proxy = {<span class="string">"http"</span>:<span class="string">"http://127.0.0.1:8087"</span>,<span class="string">"https"</span>:<span class="string">"https://127.0.0.1:8087"</span>}
response = requests.get(video_data[<span class="string">'youtube_url'</span>],proxies = proxy,verify=<span class="keyword">False</span>)
</pre></td></tr></table></figure><br>将最终版本的代码的第22行<code>response = requests.get(video_data[&#39;youtube_url&#39;])</code>替换成上面的代码即可。如果 Pyvideo.org 也上不了（校园网有时候就是这么抽风），就把 proxy 设成全局的，在所有调用 <code>requests.get()</code> 的地方设置 proxies 参数即可。</p>
<p>你可能会疑惑 <code>verify</code> 参数是干嘛的。这是对 HTTPS 请求做 SSL 验证的（YouTube 是 HTTPS 连接）。调用<code>requests.get()</code>的时候默认<code>verify</code>参数为<code>True</code>，就是会进行验证。如果我们通过代理爬取站点，SSL 验证一般是不会通过的，会返回 <code>[SSL: CERTIFICATE_VERIFY_FAILED]</code> 错误。所以需要将<code>verify</code>关闭。</p>
  
	</div>
		<footer class="article-footer clearfix">
<div class="article-catetags">

<div class="article-categories">
  <span></span>
  <a class="article-category-link" href="/categories/Python/">Python</a>
</div>


  <div class="article-tags">
  
  <span></span> <a href="/tags/Python/">Python</a><a href="/tags/爬虫/">爬虫</a><a href="/tags/译文/">译文</a>
  </div>

</div>



	<div class="article-share" id="share">
	
	  <div data-url="http://wuchong.me/blog/2014/04/24/easy-web-scraping-with-python/" data-title="使用 Python 轻松抓取网页 | Jark&#39;s Blog" data-tsina="2176287895" class="share clearfix">
	  </div>
	
	</div>


</footer>

   	       
	</article>
	
<nav class="article-nav clearfix">
 
 <div class="prev" >
 <a href="/blog/2014/04/26/odps-sql-introduction/" title="阿里推荐大赛：ODPS SQL 入门">
  <strong>上一篇：</strong><br/>
  <span>
  阿里推荐大赛：ODPS SQL 入门</span>
</a>
</div>


<div class="next">
<a href="/blog/2014/04/19/recsys-cf-study/"  title="推荐系统学习：协同过滤实现">
 <strong>下一篇：</strong><br/> 
 <span>推荐系统学习：协同过滤实现
</span>
</a>
</div>

</nav>

	
<section id="comments" class="comment">
	<div class="ds-thread" data-thread-key="blog/2014/04/24/easy-web-scraping-with-python/" data-title="使用 Python 轻松抓取网页" data-url="http://wuchong.me/blog/2014/04/24/easy-web-scraping-with-python/"></div>
</section>


</div>  
      <div class="openaside"><a class="navbutton" href="#" title="显示侧边栏"></a></div>

<div id="asidepart">
<div class="closeaside"><a class="closebutton" href="#" title="隐藏侧边栏"></a></div>
<aside class="clearfix">

  
<div class="categorieslist">
	<p class="asidetitle">分类</p>
		<ul>
		
			<li><a href="/categories/Hexo/" title="Hexo">Hexo<sup>7</sup></a></li>
		
			<li><a href="/categories/LinuxUnix/" title="Linux/Unix">Linux/Unix<sup>6</sup></a></li>
		
			<li><a href="/categories/Python/" title="Python">Python<sup>3</sup></a></li>
		
			<li><a href="/categories/推荐系统/" title="推荐系统">推荐系统<sup>3</sup></a></li>
		
			<li><a href="/categories/杂项资源/" title="杂项资源">杂项资源<sup>2</sup></a></li>
		
			<li><a href="/categories/程序设计/" title="程序设计">程序设计<sup>5</sup></a></li>
		
			<li><a href="/categories/系统架构/" title="系统架构">系统架构<sup>1</sup></a></li>
		
			<li><a href="/categories/编程语言/" title="编程语言">编程语言<sup>2</sup></a></li>
		
			<li><a href="/categories/职场生涯/" title="职场生涯">职场生涯<sup>2</sup></a></li>
		
			<li><a href="/categories/随笔生活/" title="随笔生活">随笔生活<sup>5</sup></a></li>
		
		</ul>
</div>


  
<div class="tagslist">
	<p class="asidetitle">标签</p>
		<ul class="clearfix">
		
			<li><a href="/tags/Hexo/" title="Hexo">Hexo<sup>8</sup></a></li>
		
			<li><a href="/tags/面试/" title="面试">面试<sup>5</sup></a></li>
		
			<li><a href="/tags/Linux基础/" title="Linux基础">Linux基础<sup>5</sup></a></li>
		
			<li><a href="/tags/生活/" title="生活">生活<sup>4</sup></a></li>
		
			<li><a href="/tags/博客/" title="博客">博客<sup>4</sup></a></li>
		
			<li><a href="/tags/Python/" title="Python">Python<sup>4</sup></a></li>
		
			<li><a href="/tags/阿里推荐大赛/" title="阿里推荐大赛">阿里推荐大赛<sup>4</sup></a></li>
		
			<li><a href="/tags/算法/" title="算法">算法<sup>2</sup></a></li>
		
			<li><a href="/tags/爬虫/" title="爬虫">爬虫<sup>2</sup></a></li>
		
			<li><a href="/tags/ODPS/" title="ODPS">ODPS<sup>2</sup></a></li>
		
			<li><a href="/tags/教程/" title="教程">教程<sup>2</sup></a></li>
		
			<li><a href="/tags/Jacman/" title="Jacman">Jacman<sup>2</sup></a></li>
		
			<li><a href="/tags/Java/" title="Java">Java<sup>2</sup></a></li>
		
			<li><a href="/tags/文件系统/" title="文件系统">文件系统<sup>2</sup></a></li>
		
			<li><a href="/tags/启动流程/" title="启动流程">启动流程<sup>1</sup></a></li>
		
			<li><a href="/tags/权限/" title="权限">权限<sup>1</sup></a></li>
		
			<li><a href="/tags/正则表达式/" title="正则表达式">正则表达式<sup>1</sup></a></li>
		
			<li><a href="/tags/jQuery/" title="jQuery">jQuery<sup>1</sup></a></li>
		
			<li><a href="/tags/双检锁/" title="双检锁">双检锁<sup>1</sup></a></li>
		
			<li><a href="/tags/设计模式/" title="设计模式">设计模式<sup>1</sup></a></li>
		
		</ul>
</div>


  <div class="linkslist">
  <p class="asidetitle">友情链接</p>
    <ul>
        
          <li>
            <a href="https://coderq.com" target="_blank" title="码农圈">码农圈</a>
          </li>
        
          <li>
            <a href="http://zipperary.com/" target="_blank" title="Zippera&#39;s Blog">Zippera&#39;s Blog</a>
          </li>
        
          <li>
            <a href="http://hijiangtao.github.io/" target="_blank" title="Data.Blog">Data.Blog</a>
          </li>
        
    </ul>
</div>

  <div class="weiboshow">
  <p class="asidetitle">新浪微博</p>
    <iframe width="100%" height="119" class="share_self"  frameborder="0" scrolling="no" src="http://widget.weibo.com/weiboshow/index.php?language=&width=0&height=119&fansRow=2&ptype=1&speed=0&skin=9&isTitle=1&noborder=1&isWeibo=0&isFans=0&uid=2176287895&verifier=b3593ceb&dpc=1"></iframe>
</div>


  <div class="rsspart">
	<a href="/atom.xml" target="_blank" title="rss">RSS 订阅</a>
</div>

</aside>
</div>
    </div>
    <footer><div id="footer" >
	
	<div class="line">
		<span></span>
		<div class="author"></div>
	</div>
	
	
	<section class="info">
		<p> Hello,I&#39;m WuChong. For now I&#39;m a graduate student in Beijing. <br/>
			I&#39;ll share my learning experience with you at this blog.</p>
	</section>
	 
	<div class="social-font" class="clearfix">
		
		<a href="http://weibo.com/wuchong1014" target="_blank" class="icon-weibo" title="微博"></a>
		
		
		<a href="https://github.com/wuchong" target="_blank" class="icon-github" title="github"></a>
		
		
		
		<a href="https://twitter.com/jarkwu" target="_blank" class="icon-twitter" title="twitter"></a>
		
		
		
		
		
		
		<a href="https://plus.google.com/111190881341800841449?rel=author" target="_blank" class="icon-google_plus" title="Google+"></a>
		
		
	</div>
		<p class="copyright">Powered by <a href="http://zespia.tw/hexo/" target="_blank" title="hexo">hexo</a> and Theme by <a href="https://github.com/wuchong/jacman" target="_blank" title="Pacman">Jacman</a> © 2014 
		
		<a href="http://wuchong.me/about" target="_blank" title="WuChong">WuChong</a>
		
		</p>
</div>
</footer>
    <script src="/js/jquery-2.0.3.min.js"></script>
<script src="/js/jquery.imagesloaded.min.js"></script>
<script src="/js/gallery.js"></script>

<script type="text/javascript">
$(document).ready(function(){ 
  $('.navbar').click(function(){
    $('header nav').toggleClass('shownav');
  });
  var myWidth = 0;
  function getSize(){
    if( typeof( window.innerWidth ) == 'number' ) {
      myWidth = window.innerWidth;
    } else if( document.documentElement && document.documentElement.clientWidth) {
      myWidth = document.documentElement.clientWidth;
    };
  };
  var m = $('#main'),
      a = $('#asidepart'),
      c = $('.closeaside'),
      o = $('.openaside');
  $(window).resize(function(){
    getSize(); 
    if (myWidth >= 1024) {
      $('header nav').removeClass('shownav');
    }else
    {
      m.removeClass('moveMain');
      a.css('display', 'block').removeClass('fadeOut');
      o.css('display', 'none');
        
    }
  });
  c.click(function(){
    a.addClass('fadeOut').css('display', 'none');
    o.css('display', 'block').addClass('fadeIn');
    m.addClass('moveMain');
  });
  o.click(function(){
    o.css('display', 'none').removeClass('beforeFadeIn');
    a.css('display', 'block').removeClass('fadeOut').addClass('fadeIn');      
    m.removeClass('moveMain');
  });
  $(window).scroll(function(){
    o.css("top",Math.max(80,260-$(this).scrollTop()));
  });
});
</script>

<script type="text/javascript">
$(document).ready(function(){ 
  var ai = $('.article-content>iframe'),
      ae = $('.article-content>embed'),
      t  = $('#toc'),
      h  = $('article h2')
      ah = $('article h2'),
      ta = $('#toc.toc-aside'),
      o  = $('.openaside'),
      c  = $('.closeaside');
  if(ai.length>0){
    ai.wrap('<div class="video-container" />');
  };
  if(ae.length>0){
   ae.wrap('<div class="video-container" />');
  };
  if(ah.length==0){
    t.css('display','none');
  }else{
    c.click(function(){
      ta.css('display', 'block').addClass('fadeIn');
    });
    o.click(function(){
      ta.css('display', 'none');
    });
    $(window).scroll(function(){
      ta.css("top",Math.max(140,320-$(this).scrollTop()));
    });
  };
});
</script>


<script type="text/javascript">
$(document).ready(function(){ 
  var $this = $('.share'),
      url = $this.attr('data-url'),
      encodedUrl = encodeURIComponent(url),
      title = $this.attr('data-title'),
      tsina = $this.attr('data-tsina'),
      description = $this.attr('description');
  var html = [
  '<a href="#" class="overlay" id="qrcode"></a>',
  '<div class="qrcode clearfix"><span>扫描二维码分享到微信朋友圈</span><a class="qrclose" href="#nothing"></a><strong>Loading...Please wait</strong><img id="qrcode-pic" data-src="http://s.jiathis.com/qrcode.php?url=' + encodedUrl + '"/></div>',
  '<a href="https://www.facebook.com/sharer.php?u=' + encodedUrl + '" class="article-share-facebook" target="_blank" title="Facebook"></a>',
  '<a href="https://twitter.com/intent/tweet?url=' + encodedUrl + '" class="article-share-twitter" target="_blank" title="Twitter"></a>',
  '<a href="#qrcode" class="article-share-qrcode" title="微信"></a>',
  '<a href="http://widget.renren.com/dialog/share?resourceUrl=' + encodedUrl + '&srcUrl=' + encodedUrl + '&title=' + title +'" class="article-share-renren" target="_blank" title="人人"></a>',
  '<a href="http://service.weibo.com/share/share.php?title='+title+'&url='+encodedUrl +'&ralateUid='+ tsina +'&searchPic=true&style=number' +'" class="article-share-weibo" target="_blank" title="微博"></a>',
  '<span title="Share to"></span>'
  ].join('');
  $this.append(html);
  $('.article-share-qrcode').click(function(){
    var imgSrc = $('#qrcode-pic').attr('data-src');
    $('#qrcode-pic').attr('src', imgSrc);
    $('#qrcode-pic').load(function(){
        $('.qrcode strong').text(' ');
    });
  });
});     
</script>



<script type="text/javascript">
  var duoshuoQuery = {short_name:"wuchong"};
  (function() {
    var ds = document.createElement('script');
    ds.type = 'text/javascript';ds.async = true;
    ds.src = '//static.duoshuo.com/embed.js';
    ds.charset = 'UTF-8';
    (document.getElementsByTagName('head')[0] 
    || document.getElementsByTagName('body')[0]).appendChild(ds);
  })();
</script> 







<link rel="stylesheet" href="/fancybox/jquery.fancybox.css" media="screen" type="text/css">
<script src="/fancybox/jquery.fancybox.pack.js"></script>
<script type="text/javascript">
$(document).ready(function(){ 
  $('.article-content').each(function(i){
    $(this).find('img').each(function(){
      if ($(this).parent().hasClass('fancybox')) return;
      var alt = this.alt;
      if (alt) $(this).after('<span class="caption">' + alt + '</span>');
      $(this).wrap('<a href="' + this.src + '" title="' + alt + '" class="fancybox"></a>');
    });
    $(this).find('.fancybox').each(function(){
      $(this).attr('rel', 'article' + i);
    });
  });
  if($.fancybox){
    $('.fancybox').fancybox();
  }
}); 
</script>



<!-- Analytics Begin -->

<script type="text/javascript">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-46321946-2', 'wuchong.me');  
ga('send', 'pageview');
</script>





<!-- Analytics End -->

<!-- Totop Begin -->

	<div id="totop">
	<a title="返回顶部"><img src="/img/scrollup.png"/></a>
	</div>
	<script src="/js/totop.js"></script>

<!-- Totop End -->

<!-- MathJax Begin -->
<!-- mathjax config similar to math.stackexchange -->


<!-- MathJax End -->

  </body>
</html>
